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Abstract. We classify the GL V x GL 9 -orbits in the flag variety for GL v + q 
with rationally smooth closure, showing that they are all either already closed 
or are pullbacks from orbits with smooth closure in a partial flag variety. 



1. Introduction 

Let G be a complex reductive group with Borel subgroup B. The question 
of which Schubert varieties in the flag variety G/B are smooth has received a 
great deal of attention, particularly in recent years BilLakOO, BilPosOS . Less well 
studied, but very important for representation theory, are the closures of orbits in 
G/B under the action of the fixed point subgroup K := G e of G, where 9 is an 
involutive automorphism of G |LusVog83] . Such orbit closures have been called 
symmetric varieties by Springer and are studied by him in |Spr92| . In this paper 
we use his techniques to decide which symmetric varieties are smooth in the special 
case G — GL(p + q, C), K = GL(p, C) x GL(q, C). We will give a pattern avoidance 
criterion for rational smoothness, along the lines of the well-known one for rational 
smoothness of Schubert varieties in type A. We will also show that all rationally 
smooth symmetric varieties in this case are either closed orbits or pullbacks of 
smooth varieties in partial flag varieties and so in particular are smooth. In a joint 
paper with Peter Trapa, we extend the characterization of the rationally smooth 
symmetric varieties to the flag varieties for the real groups Sp(p,q) and SO*(2n) 
[McGTrOSJ . 

I would like to thank Peter Trapa for many very helpful conversations and Leticia 
Barchini for pointing out an error in an earlier version of this paper. 

2. Preliminaries 

Now let G = GL(n,C), where n = p + q, and take 9 to be conjugation by a 
diagonal matrix on G with p eigenvalues 1 and q eigenvalues — 1 , so that K = G e — 
GL(p, C) x GL(q, C). This group may also be viewed as the complexification of the 
maximal compact subgroup U(p) x U (q) of the real form U(p, q) of G. Let B be the 
subgroup of upper triangular matrices in G. As is well known, the quotient G/B 
may be identified with the set of complete flags Vq C V\ C • • • C V n in C". Let P be 
the span of the first p vectors of the standard basis of C™. Recall that if-orbits in 
G/B are parametrized by clans, which are sequences 7 = (ci, . . . , c n ) of n symbols 
Cj, each either + or — or a natural number, such that every natural number occurs 
either exactly twice in 7 or not at all |MatOsh88. Yam97], In this parametrization 
the orbit corresponding to (ci, . . . , c„) consists of flags Vq C • • • C V n for which 
the dimension of Vi D P equals the number of + signs and pairs of equal numbers 
among or, Cj, for all i between 1 and n. In particular, the number of + signs and 
pairs of equal numbers in the entire clan must be exactly p. We identify two clans 
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if they have the same signs in the same positions and pairs of equal numbers in the 
same positions (so that for example (1, +, 1, — ) is identified with (2, +,2,—), but 
not with (1, + , — , 1)). We say that the clan 7 = (ci, . . . , c„) includes the pattern 
(di, .... dm) if there are indices ii < • ■ • < i m such that the (possibly shorter) clans 
{ci 1 , . . . , Ci m ) and {d\, . . . , d m ) are identified. We say that 7 avoids (di, . . . , d m ) 
if it does not include it. If Q is a parabolic subgroup of G, corresponding to an 
arrangement of the n coordinates into blocks of consecutive coordinates (each block 
having only one coordinate if Q = B), then the quotient G/Q may be identified 
with the set of partial flags Vq C • • • C V m in C" such that the dimension of 
Vi is the sum of the sizes of the first i blocks of coordinates. Then i^-orbits in 
G/Q are likewise parametrized by clans, except that we identify two clans when- 
ever corresponding blocks of coordinates have the same signatures (number of + 
signs plus pairs of equal numbers, and similarly for — signs) and pairs of blocks 
in one clan have the same number of numbers in common as the corresponding 
pairs of blocks in the other clan. For example, if p = q = 3 and Q corresponds to 
the coordinate arrangement (1), (2,3,4,5), (6), or to the middle three roots in the 
Dynkin diagram of G, then the clans (1, +, 2, — , 2, 1) and (1, 2, 2, 3, 3, 1) are identi- 
fied. If p = q = 4 and Q corresponds to the arrangement (1), (2, 3, 4), (5, 6, 7), (8) 
then the clans (1, +, 1, — , +, 2, — , 2) and (1, — , +, 1, +, — , 2, 2) are identified, but 
(1, +, 1, — , +, 2, — , 2) and (1, +, 2, — , +, 1, — , 2) are not, as the first and second 
blocks share the number 1 in the first clan, while the first and third blocks share 
this number in the second clan. 

The above notions of pattern inclusion and avoidance are motivated by the corre- 
sponding ones for permutations in one-line notation, which Lakshmibai and Sand- 
hya used to characterize rationally smooth Schubert varieties in type A [LakSan90] 
and Billey later extended to the other classical types |Bil98j . 

i^-orbits in G/B are partially ordered by containment of their closures. On 
the level of clans, this order includes the following operations: to make an orbit 
larger, replace a pair of (not necessarily adjacent) opposite signs by a pair of equal 
numbers; or interchange a number with a sign so as to move the number farther 
away from its equal mate in the clan (and on the same side); or interchange a pair 
a, b of distinct numbers with a to the left of b, provided that the mate of a lies to the 
left of the mate of b ( |RicSpr90[ 5.12]. |Yam971 2.4]). Thus (the orbit corresponding 
to) (1, +, 1, -) lies below (1, 2, 1, 2) and (1, +, -, 1), while (1, 2, 1, 3, 2, 3) lies below 
(1,3,1,2,2,3)) but not below (1,3,1,3,2,2). (These operations include but do 
not coincide with the ones generating the Matsuki-Oshima graph [MatOsh88].) In 
particular, the closed orbits are exactly those whose clans have only signs, while 
the open orbit has clan (1, 2, . . . , q, + . . . , +, q, q — 1, . . . , 1), with p — q plus signs, 
if p > q. 

We will need a formula for the dimension of the orbit 7 corresponding to the 
clan 7 = (ci, . . . ,c„) |Yam97l 2.3], Set d^ q := \{p{p - 1) + q{q - 1)). Then dim C> 7 
is given by 

d p .q + \_. U ~ ? ~ e N : c s = c t = k for some s < i < t < j}) 




In particular, the closed orbits all have the same dimension d PiQ . 

We conclude this section by recalling and slightly generalizing the well-known 
derived functor module construction on the level of i-T-orbits. For this purpose let G 
be any complex reductive group and K its fixed points under an involution 9. Let Q 
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be any ^-stable parabolic subgroup of G, containing the ^-stable Borel subgroup B. 
If q is the Lie algebra of Q, then the orbit K ■ q identifies with a closed orbit in the 
partial flag variety G/Q |RicSpr90} 2.5]. Its preimage 7r _1 (ii' • q) in G/B under the 
natural projection n : G/B — > G/Q is the support of a derived functor module; we 
call the open orbit in this preimage a derived functor orbit |Tra05[ §1]. Its closure 
fibers smoothly via n over K ■ q with fiber the flag variety Q/B of Q (which may 
be identified with the flag variety of any Levi factor of Q), so it is smooth. More 
generally, let Oq be any if -orbit in G/Q with smooth closure Oq. The preimage 
7r _1 (Og) fibers smoothly over Oq with fiber Q/B, so is again smooth. We also 
call the open orbit O in this preimage a derived functor orbit; to avoid trivialities, 
we assume that Q ^ B in the more general setting, unless Ob is already closed in 
G/B. (Thus closed orbits in G/B are also called derived functor orbits.) 

3. Main result 

Now we can characterize the if-orbits with rationally smooth closure. 

Theorem. If the clan 7 = (ci, . . . , c„) includes one of the patterns (1, +, — , 1), 
(1, -, +, 1),(1, 2, 1, 2), (1, +, 2, 2, 1), (1, -, 2, 2, 1), (1, 2, 2, +, 1), or (1, 2, 2, -, 1), then 
the orbit 1 does not have rationally smooth closure. Otherwise 1 is a derived 
functor orbit, so that its closure is smooth. In particular, Springer's necessary con- 
dition for rational smoothness in |Spr92| is sufficient in this setting and smoothness 
and rational smoothness are equivalent. 

Proof. Suppose first that 7 includes one of the above patterns. If this pattern has 
just two equal numbers, replace them by — and +, in that order; if it includes 
two such pairs, replace the four numbers by — , +, — , +, in that order. In all seven 
cases, continue by replacing every pair a, . . . , a of equal numbers in 7 by +,..., — . 
We obtain a clan corresponding to a closed orbit O below 7 in the partial order. 
Now Springer has defined an action of the noncompact root reflections in the Weyl 
group S n on the closed orbits, sending each such orbit to a higher orbit whose clan 
has exactly two numbers; more precisely, any two opposite signs in the clan of the 
closed orbit may be replaced by a pair of equal numbers |Spr92| 3.1,4.1]. One easily 
checks that more than dim 7 — d Ptq of these reflections send O to an orbit lying 
between it and 1 , whence 7 is not rationally smooth, as claimed |Spr92[ 3.2,3.3]. 

Now suppose that 7 avoids the above patterns. Then the intervals [s, t] of indices 
s,t with c s — c t e N are such that any two of them are one contained in the 
other or disjoint. All signs lying between any pair of equal numbers in 7 are the 
same. Moreover, if a sign lies between a pair of equal numbers, then it also lies 
between every pair of equal numbers enclosed by the first pair. If 7 has a sign 
not lying between a pair of equal numbers, let Q be the parabolic subgroup of G 
corresponding to the simple roots not involving the coordinate of the sign. Deleting 
this sign from 7, we obtain one or two clans, one consisting of the coordinates of 7 to 
the left of this sign, the other of the coordinates of 7 to the right of it. By induction 
on the number of coordinates of 7, we may assume that the orbits corresponding 
to these clans have smooth closure. One computes that the image of 1 under 7r 
identifies with the closed orbit K ■ q in G/Q, with the same clan 7, and that 1 
fibers smoothly over this orbit via the projection -k : G/B — > G/Q with smooth fiber 
parametrized by the clan 7 with the sign deleted. Hence C 7 is smooth, as desired. 
If 7 consists of at least two blocks of coordinates, each flanked by pairs of equal 
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numbers not lying between any other pair of equal numbers, with no signs lying 
between the blocks, then a similar argument shows that 7 has smooth closure, 
taking Q to be the parabolic subgroup corresponding to the simple roots whose 
coordinates lie in the same block. Otherwise the first and last coordinates of 7 arc 
a pair of equal numbers. Taking Q to be the parabolic subgroup of G corresponding 
to the remaining coordinates, we now find that the image of 7 under tt is the full 
flag variety G/Q, and the restriction of tt to this orbit closure has smooth fibers, 
parametrized by the clan consisting of these coordinates of 7. Hence in all cases 
this orbit closure is smooth. If 1 is not already closed, let a = Cj = a be a pair 
of equal numbers in 7 with no numbers between them. Let Q be the parabolic 
subgroup of G corresponding to the coordinates weakly between i and j. Then the 
image of 1 under 7r is smooth in G/Q and 7 is the derived functor orbit obtained 
from this image, as desired. □ 

In future work we hope to find similar pattern avoidance criteria for ratio- 
nal smoothness of if-orbit closures in the flag varieties of other classical groups. 
There are two nonsmooth orbit closures for GL(4, K), none for SU*(A), and one for 
SU*(6). 
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